Fix loader correctness and global side-effects from PR #103 review - #121
Conversation
Agent-Logs-Url: https://github.com/pickwicksoft/pystreamapi/sessions/df7a64c6-e671-45e1-adf6-ae09cef8ec2a Co-authored-by: garlontas <70283087+garlontas@users.noreply.github.com>
Reviewer's GuideRefines JSON and XML loader behavior to fix iteration correctness and remove shared mutable state, and updates XML/YAML loader tests to match the new semantics and improve robustness. Sequence diagram for XML loader call chain with explicit flagssequenceDiagram
actor Caller
participant xml
participant _lazy_parse_xml_file
participant _lazy_parse_xml_string
participant _parse_xml_string_lazy
participant __parse_xml
participant __parse_empty_element
participant __parse_single_element
participant __parse_multiple_elements
participant LoaderUtils
Caller->>xml: xml(src, read_from_src=False, retrieve_children, cast_types, encoding)
alt read_from_src is False
xml->>LoaderUtils: validate_path(src)
xml->>_lazy_parse_xml_file: _lazy_parse_xml_file(path, encoding, retrieve_children, cast_types)
_lazy_parse_xml_file->>_lazy_parse_xml_file: open file and read xml_string
_lazy_parse_xml_file->>_parse_xml_string_lazy: _parse_xml_string_lazy(xml_string, retrieve_children, cast_types)
else read_from_src is True
xml->>_lazy_parse_xml_string: _lazy_parse_xml_string(src, retrieve_children, cast_types)
_lazy_parse_xml_string->>_parse_xml_string_lazy: _parse_xml_string_lazy(xml_string, retrieve_children, cast_types)
end
_parse_xml_string_lazy->>_parse_xml_string_lazy: root = ElementTree.fromstring(xml_string)
_parse_xml_string_lazy->>__parse_xml: __parse_xml(root, cast_types)
alt element has no children
__parse_xml->>__parse_empty_element: __parse_empty_element(element, cast_types)
alt cast_types is True
__parse_empty_element->>LoaderUtils: try_cast(element.text)
LoaderUtils-->>__parse_empty_element: cast value
else cast_types is False
__parse_empty_element-->>__parse_xml: element.text
end
__parse_xml-->>_parse_xml_string_lazy: parsed_value
else element has one child
__parse_xml->>__parse_single_element: __parse_single_element(element, cast_types)
__parse_single_element->>__parse_xml: __parse_xml(sub_element, cast_types)
__parse_xml-->>__parse_single_element: sub_item
__parse_single_element-->>_parse_xml_string_lazy: namedtuple_single
else element has multiple children
__parse_xml->>__parse_multiple_elements: __parse_multiple_elements(element, cast_types)
loop over each child e
__parse_multiple_elements->>__parse_xml: __parse_xml(e, cast_types)
__parse_xml-->>__parse_multiple_elements: parsed_child
end
__parse_multiple_elements-->>_parse_xml_string_lazy: namedtuple_multiple
end
alt retrieve_children is True
_parse_xml_string_lazy-->>Caller: yield from __flatten(parsed)
else retrieve_children is False
_parse_xml_string_lazy-->>Caller: yield parsed
end
Flow diagram for JSON loader top-level result handlingflowchart TD
A[Start JSON load] --> B[Read JSON source - file or string]
B --> C{json_string.strip is empty?}
C -->|Yes| D[Return without yielding any items]
C -->|No| E[Call jsonlib.loads with object_hook __dict_to_namedtuple]
E --> F{Is result a list?}
F -->|Yes| G[Iterate over result and yield each item]
F -->|No| H[Yield result as a single item]
G --> I[End]
H --> I[End]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
150cd58
into
bugfix/#95/loading-big-data-files-not-safe
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
TestXmlLoader.mock_xml_file, if the implementation still usescontent = content or self.file_content, thentest_xml_loader_with_empty_filewill never actually pass an empty string ('' is falsy) and will instead useself.file_content; consider switching to a sentinel check (e.g.,if content is None: content = self.file_content) so the empty-content test behaves as intended. - The JSON loader’s file and string paths now duplicate the same
strip/jsonlib.loads/isinstance(list)logic; consider extracting this into a shared helper to keep the behavior consistent and ease future changes.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `TestXmlLoader.mock_xml_file`, if the implementation still uses `content = content or self.file_content`, then `test_xml_loader_with_empty_file` will never actually pass an empty string ('' is falsy) and will instead use `self.file_content`; consider switching to a sentinel check (e.g., `if content is None: content = self.file_content`) so the empty-content test behaves as intended.
- The JSON loader’s file and string paths now duplicate the same `strip`/`jsonlib.loads`/`isinstance(list)` logic; consider extracting this into a shared helper to keep the behavior consistent and ease future changes.
## Individual Comments
### Comment 1
<location path="tests/_loaders/test_xml_loader.py" line_range="103-106" />
<code_context>
self.assertRaises(StopIteration, next, data)
def test_xml_loader_is_iterable(self):
- with self.mock_csv_file(file_content):
+ with self.mock_xml_file(file_content):
data = xml(file_path)
self.assertEqual(len(list(iter(data))), 3)
def test_xml_loader_with_empty_file(self):
</code_context>
<issue_to_address>
**suggestion (testing):** Consider adding tests for XML loader when using `read_from_src=True` with different flag combinations
With the refactor removing the global `config` and threading flags through the call chain, the `read_from_src=True` path (`_lazy_parse_xml_string`) should be covered similarly to the file-path loader.
Could you add tests that:
- Call `xml(xml_string, read_from_src=True)` with `retrieve_children` set to both `True` and `False`, checking the parsed data shape matches the existing file-based tests.
- Call `xml(xml_string, read_from_src=True, cast_types=False)` and assert numeric/boolean-like values remain strings, mirroring `test_xml_loader_no_casting`.
You can reuse the existing `file_content` XML string so both entry points (`file_path` and `read_from_src`) stay aligned without much extra test code.
Suggested implementation:
```python
def test_xml_loader_no_casting(self):
with self.mock_xml_file(file_content):
data = xml(file_path, cast_types=False)
first = next(data)
self.assertRaises(StopIteration, next, data)
def test_xml_loader_from_src_retrieve_children_true(self):
# read_from_src=True should behave the same as the file-path loader
with self.mock_xml_file(file_content):
file_data = list(xml(file_path, retrieve_children=True))
src_data = list(xml(file_content, read_from_src=True, retrieve_children=True))
self.assertEqual(src_data, file_data)
def test_xml_loader_from_src_retrieve_children_false(self):
# read_from_src=True with retrieve_children=False should mirror file-path behaviour
with self.mock_xml_file(file_content):
file_data = list(xml(file_path, retrieve_children=False))
src_data = list(xml(file_content, read_from_src=True, retrieve_children=False))
self.assertEqual(src_data, file_data)
def test_xml_loader_from_src_no_casting(self):
# read_from_src=True with cast_types=False should mirror test_xml_loader_no_casting
with self.mock_xml_file(file_content):
file_data = list(xml(file_path, cast_types=False))
src_data = list(xml(file_content, read_from_src=True, cast_types=False))
self.assertEqual(src_data, file_data)
def test_xml_loader_is_iterable(self):
with self.mock_xml_file(file_content):
data = xml(file_path)
self.assertEqual(len(list(iter(data))), 3)
def test_xml_loader_with_empty_file(self):
with self.mock_xml_file(''):
data = xml(file_path)
self.assertRaises(ParseError, next, data)
```
These changes assume:
1. The `xml` loader already accepts `read_from_src` and `retrieve_children` keyword arguments, matching the refactor you mentioned.
2. `file_content` is the XML string used in other tests in this module, and `file_path`/`mock_xml_file` are available helpers as shown.
If there are existing dedicated tests for `retrieve_children=True/False` with the file-path loader earlier in this file, these new tests now assert that the `read_from_src=True` path produces identical output to the file-based path for the same flags, without needing to duplicate structure-specific assertions.
</issue_to_address>
### Comment 2
<location path="tests/_loaders/test_yaml_loader.py" line_range="67-72" />
<code_context>
data = yaml(file_path)
self.assertIsInstance(data, GeneratorType)
+ def test_yaml_loader_with_malformed_yaml(self):
+ malformed_yaml = "key: : invalid"
+ with self.assertRaises(yaml_lib.YAMLError):
+ list(yaml(malformed_yaml, read_from_src=True))
+
def _check_extracted_data(self, data):
</code_context>
<issue_to_address>
**suggestion (testing):** Extend malformed YAML coverage to the file-based loader path
You’ve covered the `read_from_src=True` path. To keep behavior consistent with the other loader tests, please add a file-based variant that:
- Writes the same malformed YAML to a temp file (or uses the existing file-mocking helper).
- Calls `yaml(file_path)` without `read_from_src=True`.
- Asserts `yaml_lib.YAMLError` is raised when the generator is consumed.
This will verify that both string and file inputs fail the same way for invalid YAML.
```suggestion
def test_yaml_loader_with_malformed_yaml(self):
malformed_yaml = "key: : invalid"
with self.assertRaises(yaml_lib.YAMLError):
list(yaml(malformed_yaml, read_from_src=True))
def test_yaml_loader_with_malformed_yaml_file_path(self):
malformed_yaml = "key: : invalid"
with patch(PATH_EXISTS, return_value=True), \
patch(PATH_ISFILE, return_value=True), \
patch(OPEN, mock_open(read_data=malformed_yaml)):
with self.assertRaises(yaml_lib.YAMLError):
list(yaml("malformed.yaml"))
def _check_extracted_data(self, data):
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| def test_xml_loader_is_iterable(self): | ||
| with self.mock_csv_file(file_content): | ||
| with self.mock_xml_file(file_content): | ||
| data = xml(file_path) | ||
| self.assertEqual(len(list(iter(data))), 3) |
There was a problem hiding this comment.
suggestion (testing): Consider adding tests for XML loader when using read_from_src=True with different flag combinations
With the refactor removing the global config and threading flags through the call chain, the read_from_src=True path (_lazy_parse_xml_string) should be covered similarly to the file-path loader.
Could you add tests that:
- Call
xml(xml_string, read_from_src=True)withretrieve_childrenset to bothTrueandFalse, checking the parsed data shape matches the existing file-based tests. - Call
xml(xml_string, read_from_src=True, cast_types=False)and assert numeric/boolean-like values remain strings, mirroringtest_xml_loader_no_casting.
You can reuse the existing file_content XML string so both entry points (file_path and read_from_src) stay aligned without much extra test code.
Suggested implementation:
def test_xml_loader_no_casting(self):
with self.mock_xml_file(file_content):
data = xml(file_path, cast_types=False)
first = next(data)
self.assertRaises(StopIteration, next, data)
def test_xml_loader_from_src_retrieve_children_true(self):
# read_from_src=True should behave the same as the file-path loader
with self.mock_xml_file(file_content):
file_data = list(xml(file_path, retrieve_children=True))
src_data = list(xml(file_content, read_from_src=True, retrieve_children=True))
self.assertEqual(src_data, file_data)
def test_xml_loader_from_src_retrieve_children_false(self):
# read_from_src=True with retrieve_children=False should mirror file-path behaviour
with self.mock_xml_file(file_content):
file_data = list(xml(file_path, retrieve_children=False))
src_data = list(xml(file_content, read_from_src=True, retrieve_children=False))
self.assertEqual(src_data, file_data)
def test_xml_loader_from_src_no_casting(self):
# read_from_src=True with cast_types=False should mirror test_xml_loader_no_casting
with self.mock_xml_file(file_content):
file_data = list(xml(file_path, cast_types=False))
src_data = list(xml(file_content, read_from_src=True, cast_types=False))
self.assertEqual(src_data, file_data)
def test_xml_loader_is_iterable(self):
with self.mock_xml_file(file_content):
data = xml(file_path)
self.assertEqual(len(list(iter(data))), 3)
def test_xml_loader_with_empty_file(self):
with self.mock_xml_file(''):
data = xml(file_path)
self.assertRaises(ParseError, next, data)These changes assume:
- The
xmlloader already acceptsread_from_srcandretrieve_childrenkeyword arguments, matching the refactor you mentioned. file_contentis the XML string used in other tests in this module, andfile_path/mock_xml_fileare available helpers as shown.
If there are existing dedicated tests for retrieve_children=True/False with the file-path loader earlier in this file, these new tests now assert that the read_from_src=True path produces identical output to the file-based path for the same flags, without needing to duplicate structure-specific assertions.
| def test_yaml_loader_with_malformed_yaml(self): | ||
| malformed_yaml = "key: : invalid" | ||
| with self.assertRaises(yaml_lib.YAMLError): | ||
| list(yaml(malformed_yaml, read_from_src=True)) | ||
|
|
||
| def _check_extracted_data(self, data): |
There was a problem hiding this comment.
suggestion (testing): Extend malformed YAML coverage to the file-based loader path
You’ve covered the read_from_src=True path. To keep behavior consistent with the other loader tests, please add a file-based variant that:
- Writes the same malformed YAML to a temp file (or uses the existing file-mocking helper).
- Calls
yaml(file_path)withoutread_from_src=True. - Asserts
yaml_lib.YAMLErroris raised when the generator is consumed.
This will verify that both string and file inputs fail the same way for invalid YAML.
| def test_yaml_loader_with_malformed_yaml(self): | |
| malformed_yaml = "key: : invalid" | |
| with self.assertRaises(yaml_lib.YAMLError): | |
| list(yaml(malformed_yaml, read_from_src=True)) | |
| def _check_extracted_data(self, data): | |
| def test_yaml_loader_with_malformed_yaml(self): | |
| malformed_yaml = "key: : invalid" | |
| with self.assertRaises(yaml_lib.YAMLError): | |
| list(yaml(malformed_yaml, read_from_src=True)) | |
| def test_yaml_loader_with_malformed_yaml_file_path(self): | |
| malformed_yaml = "key: : invalid" | |
| with patch(PATH_EXISTS, return_value=True), \ | |
| patch(PATH_ISFILE, return_value=True), \ | |
| patch(OPEN, mock_open(read_data=malformed_yaml)): | |
| with self.assertRaises(yaml_lib.YAMLError): | |
| list(yaml("malformed.yaml")) | |
| def _check_extracted_data(self, data): |
Addresses all unresolved review comments on the lazy-loader refactor: a correctness bug in JSON parsing, a global shared-state bug in the XML loader, and test hygiene issues.
JSON loader
yield fromon non-array top-level object was wrong.json.loads(..., object_hook=...)returns a namedtuple for a top-level{}— iterating over it withyield fromunpacks its field values, not the object itself. Now checksisinstance(result, list)and yields the single object directly when the root is not an array.src == ''replaced withnot src.strip()in the file reader path.XML loader
configobject. Mutatingconfig.cast_types/config.retrieve_childrenat call time is not safe under concurrent use. Both options are now passed as explicit arguments through the full call chain (_lazy_parse_xml_file,_lazy_parse_xml_string,_parse_xml_string_lazy, and all__parse_*helpers).Tests
test_xml_loader.py: Renamedmock_csv_file→mock_xml_file; updated docstring; addedsetUpto initializeself.file_contentand avoid a latentAttributeErrorwhencontentis not passed.test_yaml_loader.py: Addedtest_yaml_loader_with_malformed_yamlassertingyaml.YAMLErroris raised for invalid input.Summary by Sourcery
Fix JSON and XML loader behavior while tightening related tests.
Bug Fixes:
retrieve_childrenandcast_typesoptions explicitly through the parsing pipeline.Enhancements:
Tests:
setUpinitializer to prevent latent attribute errors.yaml.YAMLError.